Virtual Insertion: Robust Bundle Adjustment over Long Video Sequences

Bundle Adjustment is a key process to enhance the global accuracy of the 3D camera pose and structure estimation in the framework of structure from motion over long video sequences. However, most bundle adjustment algorithms require sufficient visual feature correspondences from each camera frame to its neighboring frames in video sequences, which are hard to collect in real environments, especially for indoor realtime navigation applications. A camera may not observe enough common scene points over a long period of time due to occlusions or non-texture background such as the white walls etc.. With the use of video images as the only input, bundle adjustment will easily fail due to the constant link outage of visual landmarks in the scene. We call it the effect of “visual breaks”, and the issue of “visual breaks” has hindered the usage of bundle adjustment. It is particularly critical for sequential Structure from Motion (sSfM) applications where motion estimation is from “chaining” neighboring key frames. On the other hand, to deal with this issue of “visual breaks”, visionbased navigation systems, such as Simultaneous Localization and Mapping (SLAM), typically do not rely on the video cameras only for robustness. Different techniques have been proposed to reduce the drift caused by“visual breaks” and other sources (e.g. inaccurate calibration) by fusing non-vision sensors, such as Inertial Measurement Unit (IMU) [1], LiDAR [3] or GPS [2]. As a result, good motion measurements from non-vision sensors or motion assumptions can be obtained easily at these “visual breaks” locations. However the bundle adjustment is still not able to use the motion estimates from these techniques directly due to a missing approach to incorporate them inside the cost function during optimization. In this paper, in order to overcome the above issue, we propose a “Virtual Insertion” scheme to construct elastic virtual links on these “visual breaks” positions to fill visual landmark link outage with the measurements provided by other sensors or motion assumptions, so that all camera positions can be linked in the long video by the real or virtual scene landmarks before bundle adjustment. This way enables the traditional bundle adjustment algorithms to achieve robust large-area structure from motion over long video sequences. Specifically, with the measurements from non-vision sensors at the “visual break” positions, we actually convert them into a set of virtual landmark links that will serve as 3D-2D projection constraints in the cost function of bundle adjustment optimization. As a result, measurements from other sensors can be integrated into existing bundle adjustment framework. Experiments on real-world long video sequences show that the virtual insertion scheme can significantly enhance both robustness and global accuracy of bundle adjustment over long video sequences in challenging real-world environments. A “visual break” is critical especially for sequential structure from motion, where usually a camera position has feature correspondences only with neighboring positions. With the help of IMU and Kalman filter [3], a visual odometry system is able to output reasonable and continuous poses using measurements from IMU especially at the “visual breaks”. However the measurements from IMU cannot be integrated into the framework of bundle adjustment directly, resulting large jumps and drifts. This is because "visual breaks" can severely affect the bundle adjustment, in the sense that the global-minima of the whole sequence becomes the combination of local-minimas in each of the two segments of the sequence because the transition between the two sets of locations is unconstrained. This is the reason why large jumps can be found in the output trajectory from bundle adjustment when “visual breaks” exist in the sequence. Figure 1 shows the illustration of a typical motion estimation over a video sequence with a “visual break” annotated with a red link arrow. Due to drift in the initial poses estimation, the loop does not close although the